Depth and chroma information based coalescence of real world and virtual world images

ABSTRACT

Methods and systems for selectively merging real-world objects into a virtual environment are disclosed. The method may include: receiving a first input for rendering of a virtual environment, a second input for rendering of a real-world environment, and a depth information regarding the rendering of the real-world environment; identifying at least one portion of the rendering of the real-world environment that is within a depth range and differentiable from a predetermined background; generating a merged rendering including the at least one portion of the rendering of the real-world environment into the rendering of the virtual environment; and displaying the merged rendering to a user.

BACKGROUND

A virtual world (may also be referred to as virtual reality, virtualenvironment, or synthetic environment) is a computer-based simulatedenvironment. Virtual reality is often used for various types of gamingas well as training purposes. For instance, a fully immersive virtualreality that uses a head-mounted display (HMD) allows a user to trainthrough a wide variety of terrains, situations and scenarios. Virtualreality may also be used in flight simulations and various othertraining operations.

SUMMARY

Embodiments of the inventive concepts disclosed herein are directed to amethod for selectively merging real-world objects into a virtualenvironment. The method may include: receiving a first input forrendering of a virtual environment, a second input for rendering of areal-world environment, and a depth information regarding the renderingof the real-world environment; identifying at least one portion of therendering of the real-world environment that is within a depth range anddifferentiable from a predetermined background; generating a mergedrendering including the at least one portion of the rendering of thereal-world environment into the rendering of the virtual environment;and displaying the merged rendering to a user.

In one aspect, embodiments of the inventive concepts disclosed hereinare directed to an apparatus. The apparatus may include at least oneinput port configured to receive a first input for rendering of avirtual environment, a second input for rendering of a real-worldenvironment, and a depth information regarding the rendering of thereal-world environment. The apparatus may also include an imageprocessor. The image process may be configured to: identify at least oneportion of the rendering of the real-world environment that is within adepth range and differentiable from a predetermined background; generatea merged rendering including the at least one portion of the renderingof the real-world environment and the rendering of the virtualenvironment; and provide the merged rendering for display to a user.

In a further aspect, embodiments of the inventive concepts disclosedherein are directed to a system. The system may include an imageprocessor. The image processor may be configured to: receive a firstinput for rendering of a virtual environment, a second input forrendering of a real-world environment, and a depth information regardingthe rendering of the real-world environment; identify at least oneportion of the rendering of the real-world environment that is: within adepth range and differentiable from a predetermined background; andgenerate a merged rendering including the at least one identifiedportion of the rendering of the real-world environment and the renderingof the virtual environment. The system may also include a display devicecoupled with the image processor. The display device may be configuredto display the merged rendering to a user.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the inventive concepts disclosed and claimedherein. The accompanying drawings, which are incorporated in andconstitute a part of the specification, illustrate embodiments of theinventive concepts and together with the general description, serve toexplain the principles and features of the inventive concepts disclosedherein.

BRIEF DESCRIPTION OF THE DRAWINGS

The numerous objects and advantages of the inventive concepts disclosedherein may be better understood by those skilled in the art by referenceto the accompanying figures in which:

FIG. 1 is an illustration depicting a head mount display used by a userinside a training facility;

FIG. 2 is an illustration depicting a merged view as presented to theuser;

FIG. 3 is an illustration depicting multiple users inside the sametraining facility;

FIG. 4 is a block diagram depicting an embodiment of a system forselectively merging real-world objects into a virtual environment; and

FIG. 5 is a flow diagram depicting an embodiment of a method forselectively merging real-world objects into a virtual environment.

DETAILED DESCRIPTION

Reference will now be made in detail to exemplary embodiments of theinventive concepts disclosed herein, examples of which are illustratedin the accompanying drawings.

A head-mounted display or helmet mounted display, both abbreviated HMD,is a display device, worn on the head or as part of a helmet, that has adisplay device in front of one or both eyes of a user. HMDs may be usedfor gaming, training, and various types of simulations. Such simulationsmay cover a wide range of applications including driving, flying, combattraining, medical procedure training and more.

It is noted, however, that using a HMD may obstruct the view ofreal-world objects. For example, a soldier wearing a HMD for trainingpurposes may not be able to view his/her hands, feet, or any equipmenthe/she is physically handling in the real world. Obstruction of suchreal-world objects may result in a loss of immersion, realism, and senseof presence and may sometimes require the user to take the HMD off touse equipment in the real world. Repetitive removing and replacing theHMD induces negative training.

Embodiments of the inventive concepts disclosed herein may selectivelybring rendering of real-world objects into a virtual world. Morespecifically, video stream(s) of real-world objects as they would beviewed from the perspective of a user are selectively merged inreal-time with video stream(s) of the virtual environment that is beingpresented to the user, allowing the user to see and use real-worldobjects while being immersed in the virtual environment without needingto remove the display device (e.g., HMD). The ability to merge realworld objects with the virtual world also allows haptic and tactilefeedback to be presented to the user effectively.

Referring to FIG. 1, an illustration depicting a HMD 100 used by a user102 inside a training facility 104 is shown. The training facility 104may include a training room as depicted in FIG. 1. For purposes ofpresentation simplicity, the training facility 104 is depicted as arectangular room; it is to be understood, however, that the trainingfacility 104 may be of different sizes and/or shapes. It is also to beunderstood that the training facility 104 may be located outdoorswithout departing from the broad scope of the inventive conceptsdisclosed herein.

Embodiments of the inventive concepts disclosed herein may selectivelybring real-world objects physically located in the training facility 104into the virtual environment that is being presented to the user 102 viathe HMD 100. For example, if the training exercise requires the user 102to operate a rangefinder 106 present in the training facility 104, itmay be desirable to merge the video image of the rangefinder 106 asviewed from the perspective of the user 102 with the virtual environmentthat is being presented to the user 102. It may also be desirable tomerge video images of other real-world objects (e.g., sandbags 108present in the training facility 104) in proximity to the user 102 intothe virtual environment. However, certain real-world objects (e.g.,training room walls) located farther away from the user 102 may beexcluded and refrained from merging into the virtual environment.

FIG. 2 is an illustration depicting a merged view 116 as presented tothe user 102. It is to be understood that certain HMDs are capable ofpresenting stereo images to both eyes of the user 102; however, forpurposes of presentation simplicity, only one of such merged images isshown. It is contemplated that similar merging techniques disclosedherein may be applicable to two-dimensional, three-dimensional, stereo,and various other types of images (or video streams) without departingfrom the broad scope of the inventive concepts disclosed herein.

As depicted in FIG. 2, a virtual environment 114 is presented to theuser 102 via the HMD 100. Also presented to the user 102 via the HMD 100are the renderings of the rangefinder 106 and sandbags 108 as viewedfrom the perspective of the user 102. Providing renderings of therangefinder 106 and sandbags 108 as viewed from the perspective of theuser 102 may be implemented by placing one or more cameras on oradjacent to the HMD 100 to simulate the viewing angles of the user'seyes. Images (or video streams) obtained utilizing such cameras may thenbe processed and superimposed onto the virtual environment 114 inreal-time to produce the exemplary merged view 116 to the user 102 asshown in FIG. 2.

It is noted that physical objects that are located a certain distanceaway (e.g., training room walls) from the user 102 are automaticallyexcluded from the merged view 116 by utilizing depth informationobserved from the perspective of the user 102. Depth information may beutilized in some embodiments to determine whether a particular physicalobject should be included in or excluded from the merged view 116. Morespecifically, if a particular physical object is greater than a certaindistance away from the user 102, that particular physical object isconsidered to be outside of the depth range and can be automaticallyexcluded from the merged view 116.

It is contemplated that depth information may be obtained utilizingvarious different techniques. For instance, depth information (may alsobe referred to as depth map) may be generated in real-time usingstereoscopic cameras located on or adjacent to (generally referred to asbeing co-located with) the HMD 100. Such a depth map may also begenerated utilizing range imaging solutions such as time-of-flight,structure light, stereo triangulation or narrow depth field techniquesand the like. Additionally and/or alternatively, a depth map may begenerated in real-time using a pre-captured three-dimensional spatialdata, such as room point cloud/mesh rendered in real-time with thecurrent HMD head pose (taking into account head position and trackinginformation). It is contemplated that the depth information may beobtained utilizing other techniques not specifically mentioned abovewithout departing from the broad scope of the inventive conceptsdisclosed herein.

While utilizing the depth information as described above enablesreal-time exclusion of physical objects that are outside of the depthrange from being merged into the merged view 116, it may be desirable tofurther exclude certain real-world objects even if they are locatedwithin the depth range. For instance, it may be desirable to exclude theimages of the training room floor where the user 102 is physicallylocated. Referring back to FIG. 1, suppose that an area of the floorlabeled 112 is outside of the depth range, the area 112 is excluded fromthe merged view 116 automatically. It is noted, however, that an area ofthe floor labeled 110 may be considered to be within the depth range,and therefore, additional processing may be carried out to exclude thecorresponding images of the area 110 from the merged view 116.

In some embodiments, one or more predetermined color hues (chromaranges/keys) may be utilized to cover the objects that are inside thedepth range but are unwanted in the merged view 116. Objects that areinside the depth range but are unwanted in the merged view 116 may begenerally referred to as background information, and the techniquegenerally referred to as color keying or chroma keying may be utilizedto effectively exclude such background information. Referring to theexample shown in FIG. 1, the area of the floor labeled 110 may bepainted in a particular chroma key color, allowing it to be readilydifferentiable from the rangefinder 106 and the sandbags 108. In someembodiments, a visual indicator 118 may be provided in the merged view116 (as shown in FIG. 2) to visually indicate to the user 102 theboundary between areas 110 and 112. It is to be understood, however,that the visual indicator 118 is not required, and that the visualindicator 118 may be toggled on and off by the user as desired in someembodiments.

It is contemplated that other digital image processing techniques mayalso be utilized in addition (or alternative) to the utilization ofchroma keys. For instance, the training facility 104 may be pre-scannedto record/map objects that are a part of the training facility 104(generally referred to as the background environment). Additionalobjects introduced to the training facility 104 after the scanningprocess may then be identified as being differentiable from thepre-scanned background environment. It is contemplated that specificimplementations of digital image processing techniques utilized fordifferentiating real-world objects from a background environment mayvary without departing from the broad scope of the inventive conceptsdisclosed herein.

It is noted that while the specific implementations may vary, threebasic rules for selectively merging real-world objects physicallylocated in the training facility 104 into the virtual environment areobserved in some embodiments of the inventive concepts disclosed herein.To reiterate, the three basic rules are:

-   -   a) Objects located outside of the defined depth range are        excluded from the merged view 116;    -   b) Objects located inside the defined depth range and identified        as a part of the background (e.g., objects in a particular        chroma key color) are also excluded from the merged view 116;        and    -   c) Objects located inside the defined depth range but not        identified as a part of the background are rendered and included        in the merged view 116.

It is noted that all three basic rules are at least partially based onthe defined depth range (or the depth information in general). Utilizingthe depth information in this manner provides several advantages. Forinstance, since objects that are located outside of the defined depthrange are automatically excluded from the merged view, only a small areainside the defined depth range needs to be processed for purposes ofbackground removal. This also allows multiple users 102 tosimultaneously use the training facility 104, as shown in FIG. 3, aslong as the users 102 are at least one depth range away from each other.Conversely, it may be desirable in certain training scenarios to showmultiple users 102 within the same merged view 116, thus they would beincluded within the depth range in a training facility. It iscontemplated that the depth range may be predetermined, user-configured,training situation specific, or dynamically adjusted, or combinationsthereof. It is also contemplated that users 102 may also useomnidirectional treadmills painted in a chroma key color to provide evenmore realistic solution with run/walk capabilities in some embodiments.

It is further contemplated that the techniques for selectively mergingreal-world objects into virtual environments are not limited to combattraining applications. Similar techniques are applicable to varioustypes of flight and vehicle simulations, and may be utilized to providevisual and tactile feedbacks for users in various situations and forperforming various tasks such as driving, flying, medical proceduretraining and more.

Referring now to FIG. 4, a block diagram depicting an embodiment of asystem 400 for selectively merging real-world objects into a virtualenvironment is shown. A processor 402 is utilized for generating thevirtual environment, which may be utilized for various purposes such asgaming, training, simulation and the like. The processor 402 is incommunication with a display device 404, such as a head mount display(or HMD), which is configured to present images (may also be referred toas video streams) to one or more users. The display device 404 mayinclude one or more positional trackers configured to track headposition and movements. Alternatively/additionally, auxiliary positionaltrackers 406 may be utilized for tracking purposes.

One or more cameras are located on or adjacent to the display 404 andare configured to obtain real-world images that generally match thefield of view of the display 404 (e.g., HMD). In some embodiments, thecameras may also be configured to obtain depth information along withthe images they obtain. Alternatively (or additionally), additionalcameras or sensors may be utilized to obtain the depth information. Theobtained real-world images and depth information are jointly processedto determine whether any portions of the obtained real-world imagesshould be merged into the virtual environment or not.

In some embodiments, a dedicated image processor hardware 408 isutilized to receive input for rendering of the virtual environment(generated by the processor 402) and input for rendering of thereal-world objects (obtained from the cameras 404) and process thereceived input to generate a merged video stream. A dedicated imageprocessor 408 is utilized to minimize any latency that may be associatedwith image processing. For example, it may be desirable to reflect auser's head movement in the merged stream within 20 milliseconds afterthe movement has occurred in real-world; if the latency is more than 25milliseconds, the user may experience motion sickness or otherundesirable conditions. It is contemplated, however, that theutilization of dedicated hardware is not required, and image processingmay be performed by the processor 402 without departing from the broadscope of the inventive concepts disclosed herein.

Regardless of whether the image processor 408 is implemented as adedicated hardware or an embedded component, the purpose of the imageprocessor 408 is to selectively merge real-world objects into thevirtual environment based on the depth and chroma information aspreviously described. The merged stream may then be provided to thedisplay device 404 and presented to the user.

Referring now to FIG. 5, a flow diagram depicting an embodiment of amethod 500 for selectively merging real-world objects into a virtualenvironment is shown. In a step 502, video input signals are receivedfrom different sources. The received signals may include video input forrendering a virtual environment, video input of a real-worldenvironment, as well was the depth information/mapping of the real-worldenvironment. The received signals may be buffered and synchronized in astep 504 based on time information associated with each signals, andthen converted in a step 506 to a common format that can be subsequentlyprocessed. It is contemplated that the format may be chosen based onspecific requirement such as resolution, compression rate, availableprocessing power, as well as other factors without departing from thebroad scope of the inventive concepts disclosed herein.

In some embodiments, geometric transformation is applied to match allinput to a common two-dimensional surface perspective for processing ina step 508. Each pixel location in the two-dimensional frames may thenbe processed based on the merging rules previously defined. Morespecifically, if a pixel location corresponds to an area that isdetermined (determination step 510) to be outside of a defined depthrange, the pixel location should be rendered based on video input forrendering the virtual environment. On the other hand, if a pixellocation corresponds to an area that is determined to be within thedefined depth range, a further determination step 512 is invoked todetermine whether the camera input at this pixel location corresponds toan unwanted background image (e.g., within a chroma key range). If thecamera input at this pixel location indeed corresponds to an unwantedbackground image (e.g., the pixel is in the chroma key range), the pixellocation should be rendered based on video input for the virtualenvironment. Otherwise, the pixel location should be rendered based onthe camera input, effectively bringing real-world images into the frame.

Once it is determined which video input (virtual or real) should be usedfor each pixel location, a merged video stream may be generatedaccordingly in a step 514. It is to be understood that while the exampleabove described image processing at a pixel level, in certainembodiments, a set of adjacent pixels may be processed jointly as a unitin a similar manner. It is contemplated that the granularity of thisprocessing step may be determined based on various factors such asresolution, available processing power, as well as other factors withoutdeparting from the broad scope of the inventive concepts disclosedherein.

It is also contemplated that additional post processing effects may beapplied in a step 516. For example, effects such as nigh vision goggle,thermal imaging, as well as other types of visual effects may beintroduced. It is further contemplated that if a head mount display isused, certain HMD specific transformations may be applied and displayedin a step 518. Such transformations may include, for example, geometrylens distortions, corrections for chromatic aberrations, multisampling,resizing and the like. It is contemplated, however, that utilization ofa HMD is not required, and that the display device may be atwo-dimensional, three-dimensional, stereo, or various other types ofdisplay devices without departing from the broad scope of the inventiveconcepts disclosed herein.

It is to be understood that the present disclosure may be convenientlyimplemented in forms of a software, hardware or firmware package. Such apackage may be a computer program product which employs acomputer-readable storage medium including stored computer code which isused to program a computer to perform the disclosed function and processof the present invention. The computer-readable medium may include, butis not limited to, any type of conventional floppy disk, optical disk,CD-ROM, magnetic disk, hard disk drive, magneto-optical disk, ROM, RAM,EPROM, EEPROM, magnetic or optical card, or any other suitable media forstoring electronic instructions.

It is to be understood that embodiments of the inventive conceptsdescribed in the present disclosure are not limited to any underlyingimplementing technology. Embodiments of the inventive concepts of thepresent disclosure may be implemented utilizing any combination ofsoftware, firmware, and hardware technology and by using a variety oftechnologies without departing from the broad scope of the inventiveconcepts or without sacrificing all of their material advantages.

It is to be understood that the specific order or hierarchy of steps inthe processes disclosed is an example of exemplary approaches. It is tobe understood that the specific order or hierarchy of steps in theprocesses may be rearranged while remaining within the broad scope ofthe inventive concepts disclosed herein. The accompanying method claimspresent elements of the various steps in a sample order, and are notmeant to be limited to the specific order or hierarchy presented.

It is believed that the inventive concepts disclosed herein and many oftheir attendant advantages will be understood by the foregoingdescription, and it will be apparent that various changes may be made inthe form, construction, and arrangement of the components thereofwithout departing from the broad scope of the inventive concepts orwithout sacrificing all of their material advantages. The form hereinbefore described being merely an explanatory embodiment thereof, it isthe intention of the following claims to encompass and include suchchanges.

What is claimed is:
 1. A method, comprising: pre-scanning a real-worldenvironment to identify background objects in the real-worldenvironment; receiving a first input for rendering of a virtualenvironment, a second input for rendering of the real-world environment,and a depth information regarding the rendering of the real-worldenvironment; identifying at least one portion of the rendering of thereal-world environment that is in proximity to a user, is within a depthrange, and is differentiable from a chroma-based predeterminedbackground and differentiable from the background objects identified bypre-scanning the real-world environment; applying at least one geometrictransformation to transform the first input for rendering of the virtualenvironment, the second input for rendering of the real-worldenvironment, and the depth information regarding the rendering of thereal-world environment to pixel locations within two-dimensional imageframes, wherein at least a second portion of the rendering of thereal-world environment is excluded based on depth information associatedwith a portion of the pixel locations indicating that the portion of thepixel locations is beyond the depth range, wherein the at least onegeometric transformation includes a transformation to correct geometrylens distortions and a transformation to correct for chromaticaberrations; generating a merged rendering including the at least oneportion of the rendering of the real-world environment into therendering of the virtual environment and excluding the at least thesecond portion of the rendering of the real-world environment from therendering of the virtual environment, the excluding being based on theat least the second portion of the rendering being beyond the depthrange and not in proximity to the user; and displaying the mergedrendering to the user.
 2. The method of claim 1, wherein the secondinput for rendering of the real-world environment is obtained from aperspective of the user and the depth range is defined as a distancemeasurement that is measured with respect to the user.
 3. The method ofclaim 1, wherein at least one chroma key is utilized to facilitate theidentification of the at least one portion of the rendering of thereal-world environment that is differentiable from the chroma-basedpredetermined background.
 4. The method of claim 1, further comprising:synchronizing the first input for rendering of the virtual environment,the second input for rendering of the real-world environment, and thedepth information regarding the rendering of the real-world environmentbased on time; and including at least a third portion of the renderingof the real-world environment into the rendering of the virtualenvironment, wherein the at least a third portion includes first objectsfrom the real-world environment at a first time that are differentiatedfrom second objects in the real-world environment at a second time, thefirst time and the second time being utilized to facilitate theidentification of the included first objects of the at least the thirdportion of the rendering of the real-world environment.
 5. The method ofclaim 1, further comprising: converting the first input for rendering ofthe virtual environment, the second input for rendering of thereal-world environment, and the depth information regarding therendering of the real-world environment to a common format for imageprocessing.
 6. The method of claim 1, wherein the rendering of thevirtual environment and the rendering of the real-world environment arestereo video renderings formatted for being presented utilizing ahead-mounted display.
 7. An apparatus, comprising: at least one inputport configured to receive a first input for rendering of a virtualenvironment, a second input for rendering of a real-world environment,and a depth information regarding the rendering of the real-worldenvironment; and an image processor configured to perform the followingsteps: pre-scan the real-world environment to identify backgroundobjects in the real-world environment; identify at least one portion ofthe rendering of the real-world environment that is in proximity to auser, is within a depth range, and is differentiable from a chroma-basedpredetermined background and differentiable from the background objectsidentified by pre-scanning the real-world environment; apply at leastone geometric transformation to transform the first input for renderingof the virtual environment, the second input for rendering of thereal-world environment, and the depth information regarding therendering of the real-world environment to pixel locations withintwo-dimensional image frames, wherein at least a second portion of therendering of the real-world environment is excluded based on depthinformation associated with a portion of the pixel locations indicatingthat the portion of the pixel locations is beyond the depth range,wherein the at least one geometric transformation includes at least oneof a transformation to correct geometry lens distortions; generate amerged rendering including the at least one portion of the rendering ofthe real-world environment and the rendering of the virtual environmentand excluding the at least the second portion of the rendering of thereal-world environment from the rendering of the virtual environment,the excluding being based on the at least the second portion of therendering being beyond the depth range and not in proximity to the user;and provide the merged rendering for display to the user.
 8. Theapparatus of claim 7, further comprising: an output port configured toprovide the merged rendering to a display device.
 9. The apparatus ofclaim 8, wherein the display device includes a head-mounted display, andwherein the merged rendering is a stereo video rendering formatted forbeing presented utilizing the head-mounted display.
 10. The apparatus ofclaim 9, wherein the second input for rendering of the real-worldenvironment is obtained from at least one camera co-located with thedisplay device.
 11. The apparatus of claim 7, wherein the rendering ofthe virtual environment and the rendering of the real-world environmentare video streams.
 12. The apparatus of claim 7, wherein the imageprocessor is further configured to: remove at least a third portion ofthe rendering of the real world environment, wherein the chroma-basedpredetermined background includes at least one predetermined color hueand the at least a third portion is removed based on the at least onepredetermined color hue.
 13. The apparatus of claim 7, wherein the atleast one geometric transformation further includes a transformation tocorrect for chromatic aberrations, multisampling, and resizing.
 14. Asystem, comprising: an image processor configured to: pre-scan areal-world environment to identify background objects in the real-worldenvironment; receive a first input for rendering of a virtualenvironment, a second input for rendering of the real-world environment,and a depth information regarding the rendering of the real-worldenvironment and differentiable from the background objects identified bypre-scanning the real-world environment; identify at least one portionof the rendering of the real-world environment that is: in proximity toa user, within a depth range, and differentiable from a chroma-basedpredetermined background; apply at least one geometric transformation totransform the first input for rendering of the virtual environment, thesecond input for rendering of the real-world environment, and the depthinformation regarding the rendering of the real-world environment topixel locations within two-dimensional image frames, wherein at least asecond portion of the rendering of the real-world environment isexcluded based on depth information associated with a portion of thepixel locations indicating that the portion of the pixel locations isbeyond the depth range, wherein the at least one geometrictransformation includes a transformation to correct for chromaticaberrations; and generate a merged rendering including the at least oneidentified portion of the rendering of the real-world environment andthe rendering of the virtual environment and excluding the at least thesecond portion of the rendering of the real-world environment from therendering of the virtual environment, the excluding being based on theat least the second portion of the rendering being beyond the depthrange and not in proximity to the user; and a display device coupledwith the image processor and configured to display the merged renderingto the user.
 15. The system of claim 14, wherein the display deviceincludes a head-mounted display device, wherein the merged rendering isa stereo video rendering suitable for being presented utilizing thehead-mounted display.
 16. The system of claim 15, wherein thehead-mounted display device is utilized to provide fully immersivetraining for the user.
 17. The system of claim 14, wherein the secondinput for rendering of the real-world environment is obtained from aperspective of the user utilizing at least one camera co-located withthe display device.
 18. The system of claim 17, wherein the depthinformation regarding the rendering of the real-world environment isobtained from the perspective of the user utilizing the at least onecamera.
 19. The system of claim 17, wherein the depth informationregarding the rendering of the real-world environment is obtained fromthe perspective of the user utilizing at least one of: a depth camera, adisparity map generated from two stereo cameras, and a set ofpre-captured three-dimensional spatial data.
 20. The system of claim 14,wherein the image processor is further configured to: remove at least athird portion of the rendering of the real world environment, whereinthe chroma-based predetermined background includes at least onepredetermined color hue and the at least a third portion is removedbased on the at least one predetermined color hue.